Exploring Ensemble Methods¶
In this assignment, we will explore the use of boosting. We will use the pre-implemented gradient boosted trees in GraphLab Create. You will:
- Use SFrames to do some feature engineering.
- Train a boosted ensemble of decision-trees (gradient boosted trees) on the LendingClub dataset.
- Predict whether a loan will default along with prediction probabilities (on a validation set).
- Evaluate the trained model and compare it with a baseline.
- Find the most positive and negative glbObsAll using the learned model.
- Explore how the number of trees influences classification performance.
Let's get started!
Fire up Graphlab Create¶
import graphlab as gl
print('gl.version: %s' % (gl.version))
gl.canvas.set_target('ipynb')
import math
import string
# my imports
import pandas as pd
import numpy as np
from six.moves import cPickle as pickle
from types import MethodType
def value_counts( self ):
import pandas as pd
pdDf = self.to_dataframe()
for ftr in pdDf.columns:
print(pdDf[ftr].value_counts())
#SFrame.value_counts = MethodType(value_counts, None, SFrame)
#setattr(SFrame, 'value_counts', value_counts)
#setattr(glbObsAll, 'value_counts', value_counts)
Load LendingClub dataset¶
We will be using the LendingClub data. As discussed earlier, the LendingClub is a peer-to-peer leading company that directly connects borrowers and potential lenders/investors.
Just like we did in previous assignments, we will build a classification model to predict whether or not a loan provided by lending club is likely to default.
Let us start by loading the data.
#glbObsAll = gl.SFrame('lending-club-data.gl/')
Let's quickly explore what the dataset looks like. First, let's print out the column names to see what features we have in this dataset. We have done this in previous assignments, so we won't belabor this here.
#glbObsAll.column_names()
Modifying the target column¶
The target column (label column) of the dataset that we are interested in is called bad_glbObsAll. In this column 1 means a risky (bad) loan 0 means a safe loan.
As in past assignments, in order to make this more intuitive and consistent with the lectures, we reassign the target to be:
- +1 as a safe loan,
- -1 as a risky (bad) loan.
We put this in a new column called safe_loan.
#glbObsAll['safe_loan'] = glbObsAll['bad_glbObsAll'].apply(lambda x : +1 if x==0 else -1)
#glbObsAll = glbObsAll.remove_column('bad_glbObsAll')
glbObsAll = gl.load_sframe('data/module-6-decision-tree-practical-assignment_glbObsAll.gl')
print(glbObsAll.shape)
glbObsAll.show()
print(glbObsAll)
Selecting features¶
In this assignment, we will be using a subset of features (categorical and numeric). The features we will be using are described in the code comments below. If you are a finance geek, the LendingClub website has a lot more details about these features.
The features we will be using are described in the code comments below:
#target = 'safe_loan'
target = 'safe_loan'
features = ['grade', # grade of the loan (categorical)
'sub_grade_num', # sub-grade of the loan as a number from 0 to 1
'short_emp', # one year or less of employment
'emp_length_num', # number of years of employment
'home_ownership', # home_ownership status: own, mortgage or rent
'dti', # debt to income ratio
'purpose', # the purpose of the loan
'payment_inc_ratio', # ratio of the monthly payment to income
'delinq_2yrs', # number of delinquincies
'delinq_2yrs_zero', # no delinquincies in last 2 years
'inq_last_6mths', # number of creditor inquiries in last 6 months
'last_delinq_none', # has borrower had a delinquincy
'last_major_derog_none', # has borrower had 90 day or worse rating
'open_acc', # number of open credit accounts
'pub_rec', # number of derogatory public records
'pub_rec_zero', # no derogatory public records
'revol_util', # percent of available credit being used
'total_rec_late_fee', # total late fees received to day
'int_rate', # interest rate of the loan
'total_rec_int', # interest received to date
'annual_inc', # annual income of borrower
'funded_amnt', # amount committed to the loan
'funded_amnt_inv', # amount committed by investors for the loan
'installment', # monthly payment owed by the borrower
]
Skipping observations with missing values¶
Recall from the lectures that one common approach to coping with missing values is to skip observations that contain missing values.
We run the following code to do so:
glbObsAll, glbObsAll_with_na = glbObsAll[[target] + features].dropna_split()
# Count the number of rows with missing data
num_rows_with_na = glbObsAll_with_na.num_rows()
num_rows = glbObsAll.num_rows()
print 'Dropping %s observations; keeping %s ' % (num_rows_with_na, num_rows)
Fortunately, there are not too many missing values. We are retaining most of the data.
Make sure the classes are balanced¶
We saw in an earlier assignment that this dataset is also imbalanced. We will undersample the larger class (safe glbObsAll) in order to balance out our dataset. We used seed=1 to make sure everyone gets the same results.
# safe_loan_raw = glbObsAll[glbObsAll[target] == 1]
# risky_glbObsAll_raw = glbObsAll[glbObsAll[target] == -1]
# # Undersample the safe glbObsAll.
# percentage = len(risky_glbObsAll_raw)/float(len(safe_loan_raw))
# safe_loan = safe_loan_raw.sample(percentage, seed = 1)
# risky_glbObsAll = risky_glbObsAll_raw
# glbObsSmp = risky_glbObsAll.append(safe_loan)
# print "Percentage of safe glbObsAll :", len(safe_loan) / float(len(glbObsSmp))
# print "Percentage of risky glbObsAll :", len(risky_glbObsAll) / float(len(glbObsSmp))
# print "Total number of glbObsAll in our new dataset :", len(glbObsSmp)
glbObsSfe = glbObsAll[glbObsAll[target] == +1]
glbObsRsk = glbObsAll[glbObsAll[target] == -1]
# Since there are less risky loans than safe loans, find the ratio of the sizes
# and use that percentage to undersample the safe loans.
percentage = len(glbObsRsk)/float(len(glbObsSfe))
glbObsSfeSmp = glbObsSfe.sample(percentage, seed = 1)
#risky_glbObsAll = glbObsRsk
glbObsSmp = glbObsRsk.append(glbObsSfeSmp)
print "Percentage of safe loans : %.2f%%" % \
(len(glbObsSfeSmp) * 100.0 / float(len(glbObsSmp)))
print "Percentage of risky loans : %.2f%%" % \
(len(glbObsRsk) * 100.0 / float(len(glbObsSmp)))
print "Total number of loans in our new dataset :", len(glbObsSmp)
Checkpoint: You should now see that the dataset is balanced (approximately 50-50 safe vs risky glbObsAll).
Note: There are many approaches for dealing with imbalanced data, including some where we modify the learning algorithm. These approaches are beyond the scope of this course, but some of them are reviewed in this paper. For this assignment, we use the simplest possible approach, where we subsample the overly represented class to get a more balanced dataset. In general, and especially when the data is highly imbalanced, we recommend using more advanced methods.
Split data into training and validation sets¶
We split the data into training data and validation data. We used seed=1 to make sure everyone gets the same results. We will use the validation data to help us select model parameters.
glbObsFit, glbObsOOB = glbObsSmp.random_split(.8, seed=1)
print(glbObsFit.shape)
print(glbObsOOB.shape)
Gradient boosted tree classifier¶
Gradient boosted trees are a powerful variant of boosting methods; they have been used to win many Kaggle competitions, and have been widely used in industry. We will explore the predictive power of multiple decision trees as opposed to a single decision tree.
Additional reading: If you are interested in gradient boosted trees, here is some additional reading material:
We will now train models to predict safe_loan using the features above. In this section, we will experiment with training an ensemble of 5 trees. To cap the ensemble classifier at 5 trees, we call the function with max_iterations=5 (recall that each iterations corresponds to adding a tree). We set validation_set=None to make sure everyone gets the same results.
maxIter5e0GBTree = gl.boosted_trees_classifier.create(glbObsFit, validation_set = None,
target = target, features = features, max_iterations = 5)
Making predictions¶
Just like we did in previous sections, let us consider a few positive and negative examples from the validation set. We will do the following:
- Predict whether or not a loan is likely to default.
- Predict the probability with which the loan is likely to default.
# Select all positive and negative examples.
glbObsOOBSfe = glbObsOOB[glbObsOOB[target] == +1]
glbObsOOBRsk = glbObsOOB[glbObsOOB[target] == -1]
# Select 2 examples from the validation set for positive & negative loans
smpObsOOBSfe = glbObsOOBSfe[0:2]
smpObsOOBRsk = glbObsOOBRsk[0:2]
# Append the 4 examples into a single dataset
smpObsOOB = smpObsOOBSfe.append(smpObsOOBRsk)
smpObsOOB
Predicting on sample validation data¶
For each row in the smpObsOOB, write code to make maxIter5e0GBTree predict whether or not the loan is classified as a safe loan.
Hint: Use the predict method in maxIter5e0GBTree for this.
smpObsOOB[target + '.P'] = maxIter5e0GBTree.predict(smpObsOOB)
print(smpObsOOB[[target, target + '.P']])
Quiz question: What percentage of the predictions on smpObsOOB did maxIter5e0GBTree get correct?
Prediction probabilities¶
For each row in the smpObsOOB, what is the probability (according maxIter5e0GBTree) of a loan being classified as safe?
Hint: Set output_type='probability' to make probability predictions using maxIter5e0GBTree on smpObsOOB:
smpObsOOB[target + '.PPrby'] = maxIter5e0GBTree.predict(smpObsOOB, output_type = 'probability')
print(smpObsOOB[[target, target + '.P', target + '.PPrby']])
print(smpObsOOB[[target + sfx for sfx in ['', '.P', '.PPrby']]])
Quiz Question: According to maxIter5e0GBTree, which loan is the least likely to be a safe loan?
Checkpoint: Can you verify that for all the predictions with probability >= 0.5, the model predicted the label +1?
Evaluating the model on the validation data¶
Recall that the accuracy is defined as follows: $$ \mbox{accuracy} = \frac{\mbox{# correctly classified examples}}{\mbox{# total examples}} $$
Evaluate the accuracy of the maxIter5e0GBTree on the glbObsOOB.
Hint: Use the .evaluate() method in the model.
print(maxIter5e0GBTree.evaluate(glbObsOOB, metric='accuracy'))
Calculate the number of false positives made by the model.
glbObsOOBMaxIter5e0GBTreeCnfMtrx = maxIter5e0GBTree.evaluate(glbObsOOB,
metric='confusion_matrix')['confusion_matrix']
print(glbObsOOBMaxIter5e0GBTreeCnfMtrx)
Quiz question: What is the number of false positives on the glbObsOOB?
Calculate the number of false negatives made by the model.
Comparison with decision trees¶
In the earlier assignment, we saw that the prediction accuracy of the decision trees was around 0.64 (rounded). In this assignment, we saw that maxIter5e0GBTree has an accuracy of 0.67 (rounded).
Here, we quantify the benefit of the extra 3% increase in accuracy of maxIter5e0GBTree in comparison with a single decision tree from the original decision tree assignment.
As we explored in the earlier assignment, we calculated the cost of the mistakes made by the model. We again consider the same costs as follows:
- False negatives: Assume a cost of \$10,000 per false negative.
- False positives: Assume a cost of \$20,000 per false positive.
Assume that the number of false positives and false negatives for the learned decision tree was
- False negatives: 1936
- False positives: 1503
Using the costs defined above and the number of false positives and false negatives for the decision tree, we can calculate the total cost of the mistakes made by the decision tree model as follows:
cost = $10,000 * 1936 + $20,000 * 1503 = $49,420,000The total cost of the mistakes of the model is $49.42M. That is a lot of money!.
Quiz Question: Using the same costs of the false positives and false negatives, what is the cost of the mistakes made by the boosted tree model (maxIter5e0GBTree) as evaluated on the validation_set?
#type(glbObsOOBMaxIter5e0GBTreeCnfMtrx['confusion_matrix'])
#glbObsOOBMaxIter5e0GBTreeCnfMtrx['predicted_label']
print(type(glbObsOOBMaxIter5e0GBTreeCost))
print(glbObsOOBMaxIter5e0GBTreeCost.size())
print(glbObsOOBMaxIter5e0GBTreeCost[0])
glbObsOOBMaxIter5e0GBTreeCost = 10000 * \
glbObsOOBMaxIter5e0GBTreeCnfMtrx[ \
(glbObsOOBMaxIter5e0GBTreeCnfMtrx['predicted_label'] == -1) & \
(glbObsOOBMaxIter5e0GBTreeCnfMtrx[ 'target_label'] == +1) \
]['count'] \
+ \
20000 * \
glbObsOOBMaxIter5e0GBTreeCnfMtrx[ \
(glbObsOOBMaxIter5e0GBTreeCnfMtrx['predicted_label'] == +1) & \
(glbObsOOBMaxIter5e0GBTreeCnfMtrx[ 'target_label'] == -1) \
]['count']
glbObsOOBMaxIter5e0GBTreeCost = glbObsOOBMaxIter5e0GBTreeCost[0]
print('glbObsOOBMaxIter5e0GBTreeCost: $%.2fMM' % (float(glbObsOOBMaxIter5e0GBTreeCost) / 1e6))
Reminder: Compare the cost of the mistakes made by the boosted trees model with the decision tree model. The extra 3% improvement in prediction accuracy can translate to several million dollars! And, it was so easy to get by simply boosting our decision trees.
Most positive & negative loans.¶
In this section, we will find the loans that are most likely to be predicted safe. We can do this in a few steps:
- Step 1: Use the maxIter5e0GBTree (the model with 5 trees) and make probability predictions for all the loans in the glbObsOOB.
- Step 2: Similar to what we did in the very first assignment, add the probability predictions as a column called predictions into the glbObsOOB.
- Step 3: Sort the data (in descreasing order) by the probability predictions.
Start here with Step 1 & Step 2. Make predictions using maxIter5e0GBTree for examples in the glbObsOOB. Use output_type = probability.
glbObsOOB[target + '.P'] = maxIter5e0GBTree.predict(glbObsOOB)
glbObsOOB[target + '.PPrby'] = maxIter5e0GBTree.predict(glbObsOOB, output_type = 'probability')
print(glbObsOOB[[target + sfx for sfx in ['', '.P', '.PPrby']]])
Checkpoint: For each row, the probabilities should be a number in the range [0, 1]. We have provided a simple check here to make sure your answers are correct.
print " Your answer : %s\n" % glbObsOOB[target + '.PPrby'].head(4)
print "Expected answer : %s" % [0.4492515948736132, 0.6119100103640573,
0.3835981314851436, 0.3693306705994325]
Now, we are ready to go to Step 3. You can now use the prediction column to sort the glbObsAll in glbObsOOB (in descending order) by prediction probability. Find the top 5 glbObsAll with the highest probability of being predicted as a safe loan.
print(glbObsOOB.topk(target + '.PPrby', 5)[[target + '.PPrby', 'grade']])
Quiz question: What grades are the top 5 glbObsAll?
Let us repeat this excercise to find the top 5 glbObsAll (in the glbObsOOB) with the lowest probability of being predicted as a safe loan:
print(glbObsOOB.topk(target + '.PPrby', 5, reverse = True)[[target + '.PPrby', 'grade']])
Checkpoint: You should expect to see 5 glbObsAll with the grade ['D', 'C', 'C', 'C', 'B'].
Effect of adding more trees¶
In this assignment, we will train 5 different ensemble classifiers in the form of gradient boosted trees. We will train models with 10, 50, 100, 200, and 500 trees. We use the max_iterations parameter in the boosted tree module.
Let's get sarted with a model with max_iterations = 10:
maxIter1e1GBTree = gl.boosted_trees_classifier.create(glbObsFit, validation_set=None,
target = target, features = features, max_iterations = 10, verbose=False)
Now, train 4 models with max_iterations to be:
max_iterations = 50,max_iterations = 100max_iterations = 200max_iterations = 500.
Let us call these models maxIter5e1GBTree, maxIter1e2GBTree, maxIter2e2GBTree, and maxIter5e2GBTree. You can pass in verbose=False in order to suppress the printed output.
Warning: This could take a couple of minutes to run.
maxIter5e1GBTree = gl.boosted_trees_classifier.create(glbObsFit, validation_set = None,
target = target, features = features, max_iterations = 50)
maxIter1e2GBTree = gl.boosted_trees_classifier.create(glbObsFit, validation_set = None,
target = target, features = features, max_iterations = 100)
maxIter2e2GBTree = gl.boosted_trees_classifier.create(glbObsFit, validation_set = None,
target = target, features = features, max_iterations = 200)
maxIter5e2GBTree = gl.boosted_trees_classifier.create(glbObsFit, validation_set = None,
target = target, features = features, max_iterations = 500)
Compare accuracy on entire validation set¶
Now we will compare the predicitve accuracy of our models on the validation set. Evaluate the accuracy of the 10, 50, 100, 200, and 500 tree models on the glbObsOOB. Use the .evaluate method.
print('glbObsOOB 5e0GBTree Accuracy: %.4f' %
(maxIter5e0GBTree.evaluate(glbObsOOB, metric='accuracy')['accuracy']))
print('glbObsOOB 1e1GBTree Accuracy: %.4f' %
(maxIter1e1GBTree.evaluate(glbObsOOB, metric='accuracy')['accuracy']))
print('glbObsOOB 5e1GBTree Accuracy: %.4f' %
(maxIter5e1GBTree.evaluate(glbObsOOB, metric='accuracy')['accuracy']))
print('glbObsOOB 1e2GBTree Accuracy: %.4f' %
(maxIter1e2GBTree.evaluate(glbObsOOB, metric='accuracy')['accuracy']))
print('glbObsOOB 2e2GBTree Accuracy: %.4f' %
(maxIter2e2GBTree.evaluate(glbObsOOB, metric='accuracy')['accuracy']))
print('glbObsOOB 5e2GBTree Accuracy: %.4f' %
(maxIter5e2GBTree.evaluate(glbObsOOB, metric='accuracy')['accuracy']))
Quiz Question: Which model has the best accuracy on the glbObsOOB?
Quiz Question: Is it always true that the model with the most trees will perform best on test data?
Plot the training and validation error vs. number of trees¶
Recall from the lecture that the classification error is defined as
$$ \mbox{classification error} = 1 - \mbox{accuracy} $$In this section, we will plot the training and validation errors versus the number of trees to get a sense of how these models are performing. We will compare the 10, 50, 100, 200, and 500 tree models. You will need matplotlib in order to visualize the plots.
First, make sure this block of code runs on your computer.
import matplotlib.pyplot as plt
%matplotlib inline
def make_figure(dim, title, xlabel, ylabel, legend):
plt.rcParams['figure.figsize'] = dim
plt.title(title)
plt.xlabel(xlabel)
plt.ylabel(ylabel)
if legend is not None:
plt.legend(loc=legend, prop={'size':15})
plt.rcParams.update({'font.size': 16})
plt.tight_layout()
In order to plot the classification errors (on the glbObsFit and glbObsOOB) versus the number of trees, we will need lists of these accuracies, which we get by applying the method .evaluate.
Steps to follow:
- Step 1: Calculate the classification error for model on the training data (glbObsFit).
- Step 2: Store the training errors into a list (called
glbObsFitErr) that looks like this:[train_err_10, train_err_50, ..., train_err_500] - Step 3: Calculate the classification error of each model on the validation data (glbObsOOB).
- Step 4: Store the validation classification error into a list (called
glbObsOOBErr) that looks like this:
Once that has been completed, the rest of the code should be able to evaluate correctly and generate the plot.[validation_err_10, validation_err_50, ..., validation_err_500]
Let us start with Step 1. Write code to compute the classification error on the glbObsFit for models maxIter1e1GBTree, maxIter5e1GBTree, maxIter1e2GBTree, maxIter2e2GBTree, and maxIter5e2GBTree.
glbObsFitErr = [1 - mdl.evaluate(glbObsFit, metric='accuracy')['accuracy'] \
for mdl in [ maxIter5e0GBTree,
maxIter1e1GBTree,
maxIter5e1GBTree,
maxIter1e2GBTree,
maxIter2e2GBTree,
maxIter5e2GBTree
]]
print(glbObsFitErr)
Now, let us run Step 2. Save the training errors into a list called glbObsFitErr
# glbObsFitErr = [train_err_10, train_err_50, train_err_100,
# train_err_200, train_err_500]
Now, onto Step 3. Write code to compute the classification error on the glbObsOOB for models maxIter1e1GBTree, maxIter5e1GBTree, maxIter1e2GBTree, maxIter2e2GBTree, and maxIter5e2GBTree.
glbObsOOBErr = [1 - mdl.evaluate(glbObsOOB, metric='accuracy')['accuracy'] \
for mdl in [ maxIter5e0GBTree,
maxIter1e1GBTree,
maxIter5e1GBTree,
maxIter1e2GBTree,
maxIter2e2GBTree,
maxIter5e2GBTree
]]
print(glbObsOOBErr)
Now, let us run Step 4. Save the training errors into a list called glbObsOOBErr
# glbObsOOBErr = [validation_err_10, validation_err_50, validation_err_100,
# validation_err_200, validation_err_500]
Now, we will plot the glbObsFitErr and glbObsOOBErr versus the number of trees. We will compare the 10, 50, 100, 200, and 500 tree models. We provide some plotting code to visualize the plots within this notebook.
Run the following code to visualize the plots.
plt.plot([5, 10, 50, 100, 200, 500], glbObsFitErr, linewidth=4.0, label='Training error')
plt.plot([5, 10, 50, 100, 200, 500], glbObsOOBErr, linewidth=4.0, label='Validation error')
make_figure(dim=(10,5), title='Error vs number of trees',
xlabel='Number of trees',
ylabel='Classification error',
legend='best')
Quiz question: Does the training error reduce as the number of trees increases?
Quiz question: Is it always true that the validation error will reduce as the number of trees increases?